Submodular-Bregman and the Lovász-Bregman Divergences with Applications: Extended Version

We introduce a class of discrete divergences on sets (equivalently binary vectors)that we call the submodular-Bregman divergences. We consider two kinds ofsubmodular Bregman divergence, defined either from tight modular upper or tightmodular lower bounds of a submodular function. We show that the properties ofthese divergences are analogous to the (standard continuous) Bregman divergence.We demonstrate how the submodular Bregman divergences generalize manyuseful divergences, including the weighted Hamming distance, squared weightedHamming, weighted precision, recall, conditional mutual information, and ageneralized KL-divergence on sets. We also show that the generalized Bregmandivergence on the Lovász extension of a submodular function, which we call theLovász-Bregman divergence, is a continuous extension of a submodular Bregmandivergence. We point out a number of applications of the submodular Bregman andthe Lovász Bregman divergences, and in particular show that a proximal algorithmdefined through the submodular Bregman divergence provides a framework formany mirror-descent style algorithms related to submodular function optimization.We also show that a generalization of the k-means algorithm using the LovászBregman divergence is natural in clustering scenarios where ordering is important.A unique property of this algorithm is that computing the mean ordering isextremely efficient unlike other order based distance measures. Finally we providea clustering framework for the submodular Bregman, and we derive fast algorithmsfor clustering sets of binary vectors (equivalently sets of sets).